cc @ArthurZucker @itazap

This has been an issue since 5.3.0 I believe, would be great to get this fixed.
Thank you @subin9

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the review @itazap! (and thanks @qflen for the redirect 🙏)

Thanks for flagging the redundant work — you're right that _tokenizer_json_is_byte_level fetches + fully parses tokenizer.json, and then convert_to_native_format parses it again right after. I dug into moving this to an existence check like _has_tekken_tokenizer_file, and here's where I landed:

On the pure tokenizer.model existence check: it's cheaper, but it's a heuristic proxy rather than an authoritative signal, so a straight swap has a couple of correctness edges:

• A byte-level Llama-3 repo that happens to ship a root tokenizer.model (community re-uploads, save_pretrained artifacts) would be misread as SentencePiece → routed to LlamaTokenizer → the exact space-dropping bug this PR fixes comes back.
• A SentencePiece Llama-1/2 fine-tune that only kept tokenizer.json (no tokenizer.model) would be misread as byte-level → behavior change for models that currently work.
• The current check is conservative (only diverts on positive byte-level detection); an absence-based check diverts much more broadly.

Note the official meta-llama/Meta-Llama-3 repos keep .model under original/, so root existence works for them — but "Llama-3 never ships tokenizer.model" isn't reliably true across the ecosystem.

What I'd propose instead — a hybrid that keeps the authoritative decision but kills the "parse on every llama load" cost:

if (
    tokenizer_auto_map is None
    and TokenizersBackend is not None
    and config_model_type in _DUAL_SCHEME_MODEL_TYPES
    and not has_file(pretrained_model_name_or_path, "tokenizer.model", **kwargs)  # SP Llama-1/2 short-circuits, no parse
    and _tokenizer_json_is_byte_level(pretrained_model_name_or_path, **kwargs)     # confirm byte-level by content
):
    return TokenizersBackend.from_pretrained(...)

So Llama-1/2 loads only do a cheap has_file HEAD and never parse, while byte-level routing stays content-authoritative. I'll match the subfolder handling from _has_tekken_tokenizer_file.

Separately, I can drop the double-parse by threading the already-parsed json (or resolved path) into convert_to_native_format if you think that's worth doing here.

Does the hybrid sound reasonable to you, or would you prefer the simpler existence check and accept the edge cases?

@itazap the 7 red checks look unrelated to this PR (which only touches tokenization_auto.py):

• tests/cli/* → 'HFCliTyperGroup' object has no attribute '_add_completion', the known huggingface_hub 1.22.0 breakage tracked in tests/cli fail with huggingface_hub 1.22.0: 'HFCliTyperGroup' object has no attribute '_add_completion': #47059.
• seed_oss::test_tp_forward → TP vs non-TP numeric flakiness (max diff 0.23).

Both came in with the main merge, not the diff — could you re-run once main is green? Thanks!

Following up on my own proposal above, @itazap — I tested the hybrid before pushing it, and it doesn't hold. The has_file short-circuit is an AND-gate, so it skips the content check in exactly the case the content check exists for.

The edge I described above as hypothetical turns out to be real: benyamini/DeepSeek-R1-Distill-Llama-8B-AWQ-w4g128 is byte-level, ships a root tokenizer.model, and declares LlamaTokenizer — quantization pipelines copy that file over from the SentencePiece checkpoint they were derived from. Round-trip on "Hello world. I'm an AI.":

I also mis-diagnosed the cost above: it's one Hub round-trip, not the parse. Warm cache on DeepSeek-R1-Distill-Llama-8B — has_file 197ms, cached_file 209ms, json.load 12–61ms. So the existence check saves ~5% and can't see the case above, while the hybrid is the slowest of the three (two round-trips). The one genuinely redundant piece is the second parse in convert_to_native_format (~60ms); happy to thread the resolved json through if you'd like that gone.

So I'd keep the routing logic as it stands. I've pushed a no-network regression test pinning the case above — it fails on main ('Helloworld' != 'Hello world') and passes here. CI is green.

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto

CI recap

Dashboard: View test results in Grafana
Latest run: 29201034798:2
Result: success | Jobs: 15 | Tests: 170,766 | Failures: 0 | Duration: 14h 31m